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PREFACE 


The Agriculture and Resources Inventory Surveys Through Aerospace Remote 
Sensing is a program of research, development, evaluation, and application of 
aerospace remote sensing for agricultural resources. This program is a coop- 
erative effort of the National Aeronautics and Space Administration, the U.S. 
Departments of Agriculture, Commerce, and the Interior, and the U.S. Agency 
for International Development. 

The work which is the subject of this document was performed by the Earth 
Resources Applications Division, Space and Life Sciences Directorate, Lyndon 
B. Johnson Space Center, National Aeronautics and Space Administration and the 
Lockheed Engineering and Management Services Company, Inc. Tasks performed by 
Lockheed were accomplished under Contract NAS 9-15800. 
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EXECUTIVE SUMMARY 


The 1980 U.S, /Canada Wheat and Barley Exploratory Experiment was designed to 
further develop state-of-the-art area estimation technology and test it in a 
foreign similar environment. 

Research, which was performed prior to the Agriculture and Resources Inventory 
Surveys Through Aerospace Remote Sensing (AgRISTARS) project, had identified 
technical issues (1) in the reliability and efficiency of estimating spring 
small grains in the U.S, Northern Great Plains and Canada and (2) in the sepa- 
ration of spring. wheat and spring barley by using remote sensing data. Appro- 
aches had been developed that provided potential improvement for solving the 
identified technical issues. Thus, the 1980 U.S, /Canada Wheat and Barley 
Exploratory Experiment was oriented toward developing and testing these appro- 
aches for potential further testing and development leading to foreign appli- 
cation. Developmental activities were initiated to produce an advanced tech- 
nology whHh was not only accurate but also efficient and objective.^ The 
improvai'.iSf'iV.s were directed toward developing an automated area estimation 
tech?5ifiiogy, with minimal analyst interaction, as one component of a foreign 
commodity production forecasting system. 

In response to these objectives, the Inventory Technology Development (ITD)* 
and Supporting Research projects developed improved crop identification pro- 
cedures, machine processing techniques, and crop calendar models. The ITD 
project integrated this technology into the area estimation system and imple- 
mented the exploratory test and evaluation. The exploratory evaluation was 
conducted in order to better understand the performance of this newly developed 
technology before proceeding to a pilot experiment for evaluation under a 
larger and more varied set of agricultural and environmental conditions. 


*The ITD was formerly cal led the t^oreign Commodity Production forecasting 
(FCPF) project. 
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The techniques developed and Integrated into the ITD developinental area esti- 
mation component for evaluation during the 1980 U^S. /Canada Wheat and Barley 
Exploratory Experiment were: (1) objective crop Identification procedures 

designed to produce consistent and accurate spring small grains identifica- 
tion/labeling results, (2) advanced machine processing techniques developed to 
improve the estimation of crop area within the sample segments (5x6 n.mi. 
areas), and (3) recently developed crop calendar models designed to provide 
improved estimates of the crop development stages for wheat and barley. 

The results of the experiment indicated that the new crop identification 
procedures performed well for spring small grains and they are conducive to 
automation. The performance of the machine processing techniques shows a 
significant improvement over previously evaluated technology. However, the 
crop calendars will require additional development and refinements prior to 
integration into automated area estimation technology. 

The evaluation has shown that the integrated technology is capable of produc- 
ing accurate and consistent spring small grains proportion estimates. How- 
ever, barley proportion estimation technology was not satisfactorily evaluated. 
Landsat sample segment data were not available for the high-density barley 
which is of primary importance in foreign regions. The low-density segments 
examined were judged as not giving indicative or unequivocal results. 

It is concluded that, generally, the spring small grains technology is ready 
for evaluation in a pilot experiment focusing on sensitivity analyses'to a 
variety of agricultural and meteorological conditions representati ve of the 
global environment. It is further concluded that a strong potential exists 
for establishing a highly efficient technology for spring small grains. 

The information in this Executive Summary is based on the following document: 

Payne, R. W. , 1980 U.S. /Canada Wheat and Barley Exploratory Experiment Summary 
Report, NASA/JSC-17406, LEMSCO-16921, July 1981. 
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1. INTRODUCTION 


1.1 PURPOSE AND SCOPE 

The purpose of this report is to present the results from the 1980 U.S. /Canada 
Wheat and Barley Exploratory Experiment. 

The developmental activities and experiments reported in this document cover 
activities of the Agriculture and Resources Inventory Surveys Through Aero- 
space Remote Sensing (AgRISTARS) Inventory Technology Development (ITD)* 
project. These activities include component-level exploratory development, 
integration and testing of crop identification procedures, alternative 
computer classification techniques, and candidate crop development stage 
models. Remote sensing research related to wheat and barley has also been 
conducted by the Environmental Research Institute of Michigan (ERIM) for the 
AgRISTARS Supporting Research project and is reported elsewhere (ref. 1). 

' 1*2 AgRISTARS PROGRAM 

The AgRISTARS program is a 6-year program of research, development, and evalu- 
ation of the application of aerospace remote sensing to monitoring agricul- 
tural resources. The program began in fiscal year (FY) 1980. The AgRISTARS 
program is a cooperative effort of the National Aeronautics and Space Admin- 
istration (NASA), the U.S. Departments of Agriculture, Commerce, and the 
Interior (USDA, USDC, and USDI), and the U.S. Agency for International 
Development (AID). The goal of this program is to determine the usefulness, 
cost, and extent to which aerospace remote sensing data can be used by the 
USDA to improve the objectivity, reliability, and timeliness of information 
required to carry out USDA missions (ref. 2). 


*The ITD project was formerly called the Foreign Commodity Production 
Forecasting (FCPF) project. 
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1.3 ITD PROJECT 


An objective of the ITD project is to develop and test procedures for using 
aerospace remote sensing technology to provide more objective, timely, and 
reliable crop production forecasting in foreign areas. To develop technology 
for use in foreign areas, the ITD project builds upon existing remote sensing 
technology and extends this technology to additional crops and regions. 

During FY 1980, two exploratory experiments were performed using U.S, data to 
develop and evaluate techniques. These experiments were the U.S. /Canada Wheat 
and Barley Exploratory Experiment (ref. 3) and the U.S. Corn/Soybean Explo- 
ratory Experiment (ref. 4). The results from the U^-S. /Canada Wheat and Barley 
Exploratory Experiment are presented in this report (refs. 3 and 5). 

Conclusions, results, and technical approaches in this report are described 
more fully in the addenda to this report (the 1980 U.S. /Canada Wheat and 
Barley Exploratory Experiment Final Report - Addenda, Volume II). 

1.4 U.S. /CANADA WHEAT AND BARLEY EXPLORATORY EXPERIMENT 

The overall objective of the 1980 U.S. /Canada Wheat and Barley Exploratory 
Experiment was to develop, test, and evaluate state-of-the-art technology for 
spring small grains, spring wheat, and barley in order to establish a basis 
for further development of the estimation technology to be applied in foreign 
regions, specifically the U.S.S.R. and, indirectly, Australia and Argentina. 
For this exploratory experiment, the technical emphasis was: 

a. To develop accurate and objective crop identification/labeling techniques 
(ref. 5). 

b. To develop a machine processing technology with improved performance 
characteristics (ref. 6). 

c. To develop alternative crop calendar/crop development stage models for 
making improved estimates of wheat and barley development (ref. 7). 
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2. EXPERIMENT DESCRIPTION 


Three tests were performed as part of the U.S. /Canada Wheat and Barley 
Exploratory Experiment (ref. 3). The first test was the labeling procedures; 
the second was the evaluation of machine processing/classification technology; 
and the third was the crop calendar/crop development stage models test. 

Figure 2-1 is the functional flow of a conceptual system which has these 
components incorporated into it. 

2.1 LABELING PROCEDURES TEST - SUMMARY DESCRIPTION 

The labeling procedures test was designed to test and evaluate a newly devel- 
oped objective labeling procedure (SSG-1). The test was conducted in two 
phases. 

a. Phase 1 - A shakedown test using six 1978 segments 

b. Phase 2 - An expanded test using 35 segments from a different 

crop year (1979) 

Locations of the segments used in the test are shown in figures 2-2 and 2-3. 
The objectives of this test were: 

a. To determine the accuracy and objectivity of the newly developed spring 
small grains (SSG) labeling procedure. 

b. To determine the accuracy of the barley estimation technology. 

In both phases of the test, an objective labeling procedure^ was used to label 
Landsat pixels (picture elements) in each segment. Input data to the new pro- 
cedure consisted of Landsat multi spectral scanner data, crop calendar informa- 
tion, and ancillary agronomic/meteorological data. (An example of the crop 
calendar information used in the procedure is shown in figure 2-4.) 


^Development of the Enhanced Baseline Spring Small Grains Procedure 
(Reformatted Labeling Procedure). Lockheed Dept. 644-1472, Dec. 1979 
(unpublished). 
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Figure 2 - 1 .- A functional flow of the iTD area estimation approach 








ORIGINAL 
OF POOR 








Sources: Field and Seed Crops Usual Planting and Harvest Dates. Agriculture Handbook No. 283, USDA/ESCS 
also published and unpublished USDA/ESCS statistics and state agriculture statistics. 

Figure 2-4.- North Dakota statewide historical crop calendar (ref, 8). 
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Figure 2-4.- Continued 
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Figure 2-4.- Concluded 


























The new procedure is designed to provide Increasingly detailed labeling Infor- 
mation at each step using a tree-structured decision logic (fig. 2-5). The 
first step consists of a labeling logic which 1s used to separate the pixels 
Into cropland and noncropland. The pixels labeled cropland in the first step 
are separated into spring small grains and other crops In the second step. In 
the third step, Landsat spectral aids are used for separating the spring small 
grains Into barley and other spring small grains. 

The segments In the labeling procedure test were processed, Independently, by 
two analysts In order to evaluate the repeatability and objectivity of the 
procedure. The evaluations were performed by comparing all labeling results 
to the segment ground-truth Inventories. An error characterization study was 
performed to determine If any changes to the new labeling procedure were 
required to improve the objectivity or accuracy. 

2.1.1 OBJECTIVE LABELING j>ROCEDURE - BACKGROUND AND DISCUSSION 

The objective reformatted spring small grains labeling procedure (SSG-1) 
reduces the labeling decisions to a series of steps. These steps, when 
executed in an objective manner, allow analysts with limited experience to 
follow the procedure and arrive at crop Identification labels with consis- 
tency. The development and description of this procedure are detailed in 
reference 9, 

The U.S. /Canada Wheat and Barley Exploratory Experiment (ref. 3) was jdesigned 
to evaluate the new objective procedure (ref, 5). Phase 1 (shakedown test) 
was completed in the second quarter of FY 1980. Following a critical exami- 
nation of the shakedown test results, refinements and modifications were made 
to the new objective reformatted spring small grains labeling procedure in 
preparation for additional testing which is Phase 2 (ref. 9). Two of these 
modifications are worthy of note: 

a. A method for utilizing alternate dots was incorporated into the objective 
reformatted procedure in order to increase dot purity. 
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Figure 2-5.- The major steps in the reformatted objective labeling logic 
which leads to identification of barley and other spring small grains. 
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b. Current-year crop growth stage model results and statewide historical crop 
calendars were incorporated into the procedure, (An example of a state- 
wide historical crop calendar is given in figure 2-4.) 

The expanded tests (Phase 2, Parts 1 and 2) of the experiment began in the 
third quarter of FY 1980, using the new reformatted proceaure (SSG-1) des- 
cribed in reference 9. Phase 2, Part 1, consisted of a processing of the 
segments using the older integrated procedure S3G-0 for comparison with the 
objective reformatted procedure. SSG-0 was developed during the Large Area 
Crop Inventory Experiment (LACIE) Transition Year (TY) [ref. 10]. The 
earlier integrated approach to labeling takes advantage of an analyst's 
experience and intuition. It was a learning experience, a necessary precursor 
to the eventual development of more objective labeling methodology, A gen- 
eralized functional flow of the interated procedure is shown in figure 2-6(a). 

For the integrated approach, a given segment was manually processed using the 
detailed analysis procedures developed during the LACIE and LACIE TY projects. 
This consisted of a team concept to labeling and crop signature review. The 
crop signatures and labels are then further reviewed by a quality assurance 
(QA) function. Finally, the segments are passed through a verification 
component that evaluates the estimates, in conjunction with other broad-level 
data, for trend analysis and problem detection/solving before being released 
for aggregation. vTonfidence levels using the integrated approach are directly 
related to the available Landsat acquisitions during the growing season. 

Figure 2-6(b) describes the confidence levels, from high to very low, -based on 
acquisitions used throughout the growing season for the analysis. 

2.1.2 HISTORICAL DEVELOPMENT OF THE LABELING PROCEDURES 

In support of the acreage estimation processes of the LACIE (ref. 11), agri- 
cultural analysts have relied on temporal analysis of Landsat data (ref. 12) 
to identify specific crops. Although the accuracy of identification was suf- 
ficient to provide estimates which met the goals of the experiment, consistent 
results between analysts were difficult to obtain without an intricate system- 
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Integrated/procedure- TY operations functions 



Figure 2-6(a).- Integrated analysis generalized functional flow 

(original procedure). 
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Confidence level 

Criteria description 

High 

An acquisition during each of four analysis windows: 


1. Preplant to emergence 

2. Full ground cover 

3. Ripening/ripe 

4. Harvest to postharvest 


Spring small grains dominant crop 
Medium to large fields 
Prior years' Landsat imagery 

Medium 

An acquisition during three of four analysis windows 
Previous year's Landsat data 

Low 

An acquisition in only two of four analysis windows 
Previous year's Landsat data 

Very low 

An acquisition in only one or two of four analysis windows 
Absence of previous year's Landsat data 


Figure 2-6(b)«- Confidence-level criteria for the integrated procedure. 
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of quality assurance and teamwork. Clearly, a more objective procedure was 
needed. 

Subsequent to the LACIE, developmental efforts were directed toward the expan- 
sion of the crop identification procedures to include other crops in addition 
to wheat, the principal crop of interest in the LACIE. The other crops 
initially selected for emphasis were corn and soybeans, with the U.S. Corn 
Belt as the primary area of concern. 

The procedure (refs. 13 and 14) resulting from this developmental effort 
utilized decision-tree methodology and was designed to reduce the impact of 
analyst subjectivity. The step-by-step design for corn and soybean labeling 
provided an additional benefit of allowing error sources to be isolated, thus, 
providing meaningful feedback for procedure modification. 

Because of the apparent success of this procedure with corn and soybeans, con- 
version of the U.S. spring small grains procedure to a similar format was a 
logical step. This reformatting was accomplished in preparation for the U.S./ 
Canada Wheat and Barley Exploratory Experiment. 

2.1.3 THE OBJECTIVE REFORMATTED APPROACH TO LABELING (ref. 3) 

An objective was established to develop a procedure for labeling spring small 
grains in the U.S, Northern Great Plains (USNGP) segments by converting the 
U.S. spring small grains and barley separation procedure, which was used 
during the TY project, to a format similar to the corn and soybeans decision 
logic (refs. 13 and 14). The techniques used in the TY project were to be 
enhanced whenever possible. 

Following a comprehensive review of the TY project labeling procedures 
(ref. 12), scientists identified alternative methods for performing some of 
the steps. These methods leave fewer subjective decisions in the labeling 
process to the analyst. The new techniques were tested using segments from 
the develolpmental data set,- Necessary modifications and revisions ivere made 
before incorporating them i ito the overall labeling procedure. 
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The objective labeling procedure is based primarily on analysis and observa- 
tions of the segments comprising the developmental data set. Criteria used 
for selecting the segments were based upon (a) the segments having a suffi- 
cient number of acquisitions to adequately describe the growth cycle of spring 
small grains and (b) the segments having a reasonably large proportion of 
spring small grains, particularly barley. 

Essentially three snajor divisions exist within the objective labeling proce- 
dure. These divisions are: 

1. The separation of dots (pixels) into either cropland or noncropland 

2. The separation of cropland dots into spring small grains or non -spring 
small grains. (In this document, non-spring small grains includes all 
crops other than spring small grains in addition to all nonagri cultural 
areas such as forests, water, urban areas, and bare soil.) 

3. The separation of spring-small -grain dots into barley or other spring 
small grains 

For the cropland and noncropland separation, the procedure relies on a 
slightly modified portion of the decision logic for major land-use categories, 
a part of the corn and soybeans procedure (refs. 13 and 14.) 

Because segments can be processed without an acquisition during the time when 
barley is green vegetation, the first major division had to be modified to 
ensure that barley would be labeled cropland. This modification alloy^s fields 
to be labeled cropland (provided specific conditions are satisfied) even 
though acquisitions showing the crop growing are unavailable. Additionally, 
when responses are such that the decision is clearly noncropland, the dot is 
labeled as noncropland instead of attempting a further breakdown into range, 
forest, and other. 

The successful identification of spring small grains is usually the result’ of 
an analyst *s ability to recognize fields which follow the development pattern 
of spring small grains and isolate acquisitions on which most or all" of the 
spring small grains exhibit similar characteristics (e.g., bare soil, green 
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vegetation, and harvested). If the coupling of two or more of these acquisi- 
tions provides a unique signature for spring small grains (e.g,, bare soil on 
acquisition 1 and green vegetation on acquisition 2), accurate labeling should 
result. In order to develop procedures for this process, a window technique 
was devised to select acquisitions on which the appearance of spring small 
grains would be predictable. The desired characteristics of spring small 
grains on acquisitions selected to represent each window are presented in 
table 2-1, 

If the correct acquisitions are selected, a description of the desired appear- 
ance of spring small grains, as a function of window, should allow accurate 
separation of spring small grains from non-spring small grains. In an attempt 
to provide a more objective description of appearance, green numbers and 
brightness (refs. 15, 16, and 17) were used in lieu of color descriptions for 
this procedure. 

Observation of the behavior of the green number and brightness of spring small 
grains on segments from the developmental data set was used to establish the 
green number and brightness criteria for spring small grains as a function of 
acquisition and window. These criteria cutoffs were utilized in the decision 
logic for spring small grains. 

For the separation of barley and other spring small grains, much of the TY 
project labeling procedure (ref. 1^) was retained. However, there are several 
important modifications described in addendum 1, volume II, of this document 
and are summarized below: 

a. The separation acquisition is selected using an objective procedure. This 
is the window 3 acquisiton. 

b. The decision boundary on the green number versus brightness scatter plot 
is a straight line with fixed slope. 

c. The concept of dot drift is introduced to assist in determining the loca- 
tion of the decision boundary. Dot drift is the direction of movement in- 
the green number and brightness plane from the window 2 acquisition to the 
window 3 acquisition. 
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TABLE 2-1.- DESIRED CHARACTERISTICS OF ACQUISITIONS 
AS A FUNCTION OF WINDOW 


Window 

Description of spring small grains 

Product 1 appearance of 
spring small grains 

1 

Plowing/planting for spring small 
grains 

All spring small grains appear to 
be bare soil 

Spring wheat Robertson stage 
0.8 to 2.6 

Light to dark green, 
light to dark gray, 
and black 

2 

All spring small grains appear to 
be green vegetation. (Most of the 
summer crops appear to be bare 
soi 1 . ) 

Spring wheat Robertson stage 
3.8 to 4.5 

Red, pink, brown, and 
orange 

3 

Spring barley is turned/harvested; 
spring wheat, oats, and flax appear 
to be green vegetation. 

Spring wheat Robertson stage 4.7 
to beginning of harvest. 

Deep red, reddish brown, 
brown, orange, pink, yellow, 
gold, olive, white, gray, 
and green 

4 

All spring small grains appear to 
be turned/harvested. 

Light to dark green, light 
to dark gray, white, yellow, 
gold, olive, and black 








The definition of a mlnlmutn data set for processing segments with the refor- 
matted labeling procedure reflects extensive LACIE experience in addition to 
observations of the segments from the developmental data set. 

An acquisition in window 1 was known to be a requirement In mixed wheat areas 
to provide separation between winter and spring small grains. This require- 
ment was extended to all of the areas of interest because of Its additional 
value for separating natural vegetation and hay. (See tables 2-1 and 2-2.) 

An acquisition in window 2 or window 3 is required to provide a date when 
spring small grains are growing. Since the barley separation technique relies 
on the observation of barley turning and harvested while the other spring 
small grains are in earlier stages, a window 3 acquisition is required to 
execute that portion of the procedure. 

An acquisition in window 4 is essential in areas such as South Dakota and 
Minnesota to avoid confusion of summer crops such as corn with spring small 
grains. 

A detailed description of the reformatted spring small grains labeling proce- 
dure used in Phase 2, Part 2, of the exploratory experiment is provided in- 
reference 9. The general flow of the steps involved in the procedure is shown 
in figure 2-7 of this document. A detailed description of all the steps 
required for execution of the reformatted labeling procedure is documented in 
reference 9; a summary description follows. 

When using the procedure, the analyst must first use crop calendars for spring 
wheat and barley to determine the opening and closing dates for each of the 
windows described in table 2-2 of this report. This Is Step I in figure 2-7. 
Following acquisition selection (Step I), the combination of available acqui- 
sitions and windows is considered to determine the type of labeling, if any, 
that can be performed using the procedure. If the available acquisitions and 
windows are sufficient for barley separation, the entire procedure can be 
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Figure 2-7.- The flow diagram of the reformatted spring small grains 

labeling procedure (ref. 9). 
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TABLE 2-2.- OBJECTIVE REFORMATTED LABELING PROCEDURE WINDOW DEFINITIONS USED 
IN CONJUNCTION WITH STATEWIDE HISTORICAL CROP CALENDARS^ 


Window 

Open 

Closed 

1 

Spring wheat, 50 percent 
planted minus 5 days 

Spring wheat, 50 percent 
planted plus 18 days 

2 

Spring wheats 50 percent 
headed minus 10 days 

Spring wheat, 50 percent 
headed plus 10 days 

3 

Spring barley, 50 percent 
turning to ripe minus 
6 days 

Spring barley, 50 percent 
turning to ripe plus 
6 days 

4 

Spring wheat, 50 percent 
harvested plus 15 days 

Spring wheat, 50 percent 
harvested plus 30 days 


^See figure 2-4. 
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executed. If no acquisition from window 3 is available, only the sprlng- 
small-grain portion (Steps IV through VII in fig. 2-7) of the procedure can be 
used. 


2.2 MACHINE PRQCESSING/CLASSIFICATION TEST - SUMMARY DESCRIPTION 
2.2.1 GENERAL 

The machine processing/classification test (ref. 3) consisted of processing 
and classifying the same 1979 United States spring wheat region segments used 
In the labeling procedures test. The objective of the test was to evaluate 
the accuracy and efficiency of alternative classification techniques, 

A need for more efficient use of labeled samples In segment proportion estima- 
tion had previously been established by studies which showed that simple 

random sampling could produce results equivalent to maximum likelihood classi- 
« 

fication. During the Supporting Research project, a Bayes approach to propor- 
tion estimation using a stratified sample in response to this deficiency was 
developed (ref. 18). ‘This technique was integrated with the labeling pro- 
cedure to form a proportion estimation component. It was included in the 
exploratory experiment for evaluation. 

The following alternative techniques for allocating samples and estimating 
crop proportion within each segment were evaluated (ref. 6). 

a. Random sample/relative count - this technique allocates samples randomly 
and estimates crop proportions by determining the number of samples in a 
crop category and dividing by the total number of samples. 

b. Proportional allocation/relative count - samples are allocated to clusters 
proportional to the cluster sizes, and the estimate is generated by 
determining the number of samples in a crop category per cluster and 
weighing the estimate by cluster size. 

c. Proportional allocation/Bayesian estimator - the samples are allocated to 
clusters proportional to cluster size, and proportion estimation is calcu- 
lated using the Bayesian estimator. 
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d. Bayesian sequential allocation/Bayesian estimator - samples are allocated 
to clusters sequentially In an attempt to minimize the mean square error 
(MSB), and a proportion estimate is calculated using the Bayesian 
estimator* 

In the last three evaluations, the samples were stratified using the CLASSY 
clustering algorithm (refs. 19, 20, and 21). 

2.2.2 BACKGROUND ON MACHINE PROCESSING 

Since large-area acreage estimates for small grains depend upon segment-level 
proportion estimates, It is Important that those proportion estimates be as 
accurate and precise as possible. Prior to the AgRISTARS program, several 
procedures were tested In an attempt to find an accurate and efficient method 
for estimating small -grain proportions. In the resultant method, Procedure 1 
(PI), labels were used In the random selection of training pixels to start a 
clustering algorithm. Then, cluster statistics were used to produce a maximum 
likelihood classification of. the scene Into 2- or 3-class strata. Finally, 
stratified proportion estimates were made using a second random set of labeled 
dots. However, this classification component provided no better results than 
those which could have been produced through simple random sampling. Thus, 
clustering had not been an effective method. 

Consequently, a new clustering algorithm was developed (refs. 21 and 22). The 
new algorithm used clusters to generate strata within which the crop propor- 
tions could be estimated. One advantage of this algorithm was that, -as an 
unsupervised routine, a first set of training dots was not needed (as in PI). 

In addition, a proportion estimation technique (ref. 18) which used the clus- 
ters of this algorithm was developed. This technique Involved Bayesian esti- 
mation of cluster-level proportions based on historical information concerning 
cluster purities. The cluster-level estimates were then weighted by their 
relative cluster sizes and aggregated to produce the segment-level estimate. 
Use of this technique was expected to provide better proportion estimates. 

The technique also implemented sequential sampling in an attempt to sample the 
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segment clusters more effectively and further reduce the expected of the 
proportion estimation. Characteristic of this new estimation technique was 
the sol action of dots, one at a time. It is known as the Bayesian sequential 
allocation/Bayesian estimator. The sampling technique was an attempt to 
minimize the MSE of the proportion estimate. Before each sampling of a dot, 

expected effects to MSE estimates were made for each cluster; and, on the 

basis of these estimates, a sample was taken from the cluster that was 

expected to most reduce the MSE, This manner of sampling provided an 

additional feature; the option of sampling with a fixed sample size or vary- 
ing the sample size from segment to segment. Varying the sample size could be 
managed by halting the sampling when a predetermined threshold was obtained 
for the internal MSE estimate. Varying the sample sizes in this manner was to 
provide uniform accuracy across segments by sampling more frequently from more 
"difficult" segments. 

A 10-segment development test of the Bayesian sequential allocati on/Bayesian 
estimator (ref. 23) showed that there was at least a 2 to 1 reduction in the 
MSE from that observed from PI, a reduction in proportion estimation error, 
and improved analyst labeling accuracy. Table 2-3 summarizes the proposed 
advantages of the Bayesian sequential allocation/Bayesian estimator technique 
compared to the earlier PI proportion estimation technique, 

2.3 CROP DEVELOPMENT STAGE MODEL TEST - SUMMARY DESCRIPTION 
2.3.1 GENERAL 

The crop development stage model test consisted of estimating the planting 
date and phenol ogi cal development stages of wheat and barley in 49 segments 
within the U.S, spring wheat region. Figure 2-8 shows the location of the 
segments used in the test,. 

The objectives of this test were; 
a. To evaluate alternative models. 
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• forty-nine segments in the spring wheat areas of the 
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Figure 2-8.- Segments used In the crop develop,„eut stage model test 


TABLE 2-3.- PROCEDURE 1 COMPARED TO THE BAYESIAN SEQUENTIAL 
ALLOCATION/BAYESIAN ESTIMATOR TECHNIQUE 


Step 

Procedure 

Bayesian sequential 
allocation/ 

Bayesian estimator 

Proposed advantage 

1. Stratification 

ISOCLS 

Use type 1 labeled 
dots to collapse 
clusters into two 
strata 

CLASSY 

No need to label 
dots to create a 
small number of 
strata for sampl- 
ing, thus more 
efficient 

2* Allocation of 
dots to be 
labeled 

Approximately 
proportional to 
size of strata 
(poststrati fj cation) 

Sequential to 
minimize MSE 

Requires less dots 
for same accuracy 
by incorporation 
of: 

1. Prior informa- 
tion of dis- 
tribution of 
cluster purity 

2. Knowledge of 
previously 
labeled 
samples 



• 

More accurate 
labeling for 
selected dots 

3. Stratum-1 pvel 
estimation 

Relative count 

Bayes 

Reduction in MSE 
for equivalent - 
number of dots by 
including prior 
information of 
distribution of 
cluster purity 

4. Segment-level 
estimation 

Weighted average 
over strata 

Weighted average 
over ^jtrata 

. 

None (same) 
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b. To determine which combination of planting date and phenol ogi cal develop- 
ment stage models most accurately estimate the development of wheat and 
barley. 

c. To determine if the various models are sufficiently accurate to be 
incorporated into objective labeling procedures. 

The models evaluated in this test are: 

a. Planting date models tested: 

f Normal planting date model (ref. 24) 

• Feyerherm planting date model (ref. 25) 

b. Wheat phenological development stage models tested: 

• Original Robertson wheat model (ref. 24) 

• Improved Robertson wheat model, version 1 (ref. 25) 

• Improved Robertson wheat model, version 2 (ref. 25) 

e. Barley phenological development stage model tested: 

• Williams barley model (ref. .26) 

The Feyerherm and the normal planting date models were evaluated on their 
ability to accurately predict the median planting dates in the segments. The 
basis for comparison was the ground-truth median planting dates. The ground- 
truth median planting dates for spring wheat and barley were obtained by 
calculating the date at which 50 percent of the spring wheat and barley fields 
in each of the segments were observed and planted. Discrepancies between 
ground truth and the models were measured in number of days. 

The performances of the three Robertson growth stage models were evaluated 
using the ground-truth median growth stages as the basis for comparison. 
Observed median planting dates were used to initiate the models. The ground- 
truth median growth stages for spring wheat and barley were obtained by cal- 
culating the observed median stage for spring wheat and barley fields within 
ea-^h of the segments for each of the dates on which the stages were -observed.' 
The comparison of the model's prediction versus the observed crop stage 
yielded errors in terms of crop stages associated with each of the models. 
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The barley growth stage model was evaluated using the observed median planting 
dates for barley to initiate the models and, subsequently, comparing the model 
prediction of stage with the ground-truth median growth stages for barley. 

2.3.2 DESCRIPTION OF THE GROWTH STAGE MODELS 

Robertson's concept (ref. 27) is based on certain physiological processes that 
are central to the development of spring wheat. Since temperature and photo- 
period are two primary environmental factors that influence the phenol ogi cal 
development, a photothermal concept was used to compute the development of a 
crop over five fairly short and uniform physiological periods. The tri quad- 
ratic responses of temperature and photoperiod were estimated for each of the 
phenological stages by an interative regression technique. 

The improved Robertson model, verisons 1 and 2, are improvements over the 
original Robertson model with respect to the photoperiod and temperature 
responses. The photoperiod response is limited to stages between emergence 
and flowering. The thermal response for subsequent stages are adjusted to 
represent realistic physiological responses. The development rates of spring 
wheat immediately before and after flowering are responsive primarily to the 
daily maximum temperature. 

The Williams barley model is based approximately on the same concept as the 
Robertson model; the difference is that the coefficients were developed 
specifically for barley. 

Figure 2-9 is a schematic of the model's input requirements and resultant 
output data. The normal model, although not an agrometeorological model, is 
included in figure 2-9 for the sake of completeness. It is based on historical 
data averaged for the crop reporting district. The daily minimum and maximum 
temperatures are obtained from reports of weather stations nearest the 
segments. 
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Figure 2-9.- S€tenatic of the planting date and growth stage model evaluation 






3. EXPERIMENT RESULTS 


3.1 LABELING PROCEDURES TEST RESULTS SUMMARY 

The shakedown test (Phase 1) of the new objective labeling procedure using 
1978 Landsat data indicated: 

a. Excellent spring small grains labeling accuracy results. The overall 
accuracy was 76 percent, 

b. Labeling results were comparable to those obtained from an analyst inten- 
sive procedure performed on 1978 data (76 percent versus 75 percent). 

c. Consist'?ncy between the analysts was very good. Overall, the agreement on 
labels ms 85 percent. 

The expanded labeling test (Phase 2) using the 1979 data provided the follow- 
ing results; 

a. Labeling accuracy results for spring small grains were similar to the 
integrated analysis procedure, although slightly lower, 66 percent for the 
objective labeling procedure versus 76 percent for the integrated analysis 
procedure. 

b. The 1979 error characterization study identified the areas requiring 
improvements to the objective labeling procedure (SSG-1). 

(1) The procedure processed only 25 percent of the available segments. 

(2) Confusion of pasture with small grains was a problem. 

(3) Crop calendar improvements were required in order to better select 
acquisitions for processing. 

The wheat/barley separation procedure results are: 

a. Segments with 10 percent and above in barley were not available for 
testing. 

b. Segments were not available which have both winter wheat and spring 
barley, as in the foreign similar environment. 
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c. The labeling accuracy was approximately 50 percent in low-density barley 
segments (those containing 5 percent or less). 

Consistency of labeling between two analysts labelings using the reformatted 
labeling procedure (SS6-1) averaged 78 percent (Phase 2). 

The summary given in section 3.1 is based on information in reference 3. 

3.1.1 SHAKEDOWN TEST, PHASE 1, WITH 1978 DATA - DETAILED RESULTS 

3. 1.1.1 Experiment Design for the Shakedown Test, Phase 1, 1978 Data 

In the shakedown test, all 209 dots for six segments were labeled using data 
from the 1978 crop year. The actual number of dots evaluated per segment 
varied downward from 209 because of clouds, cloud shadows, data dropouts, 
striping, or missing ground-truth inventory. The loss was a small percentage 
of the dots. Locations of the segments are shown in figure 2-2, section 2. 
Each of the segments was labeled by two analysts working independently. By 
comparing the two sets of 'labeling results, the consistency of the objective 
procedure could be evaluated. Five of these six segments were previously 
processed using the integrated labeling procedure. (Refer to addendum 1, 
volume II, of this document for details.) These labeling results were used to 
compare the accuracy of the objective reformatted labeling procedure with the 
accuracy of the integrated labeling procedure. 

3. 1.1.2 Shakedown Test, Phase I, Overall Labeling Accuracy for Final- Labels 

Table 3-1 shows the labeling accuracy for each of the categories labeled non- 
small grains, barley, and other small grains. The labeling accuracy is shown 
for all the dots labeled and for those dots which were determined by the 
analyst to be pure, mixed, or misregistered. The labeling accuracy was grea- 
ter for the pure dots (which were labeled using the decision logic) than for 
the impure dots (which were labeled by comparison with the pure dot labels.). 
The numbers in parentheses in the table show the percentage of dots correctly 
labeled when both analysts agreed on the label. The labeling accuracies were., 
in general, greater when there was agreement between the analysts. 
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TABLE 3-1.- ANALYST LABELING ACCURACY - OBJECTIVE REFORMATTED 
LABELING PROCEDURE, SHAKEDOWN TEST, PHASE I 

Note.: The numbers In parentheses show the percental? of dots 

correctly labeled when both analysts agreed on the label. 




Correctly labeled dots, % 

Crop 

category 

All 

dots 

Pure 

dots 

Mi xed 
dots 

Misregis- 
tered dots 

Nonsmall 

grains 

91 (95) 

94 (97) 

73 (78) 

82 (91) 

Small grains 
(except barley) 

72 (82) 

74 (84) 

66 (83) 

55 (63) 

Bari ey 

51 (49) 

50 (51) 

60 (-) 

50 (-) 

Total small grains 

77 (86) 

79 (87) 

73 (86) 

67 (76) 
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Table 3-2 shows a comparison, on a segment -by-segment basis, between accuracy 
obtained using the objective reformatted labeling procedure and that obtained 
using the integrated labeling procedure. Overall, the reformatted labeling 
procedure produced labeling accuracies which were comparable to the accuracies 
for the integrated labeling procedure. For some segments, the reformatted 
labeling procedure obtained better results in certain categories than did the 
integrated laieling procedure, whereas, on other segments the reverse was 
true. 


The barley labeling accuracy was not very high for either procedure, with only 
half of the barley being labeled correctly. However, the segments involved in 
this test had an average barley proportion of only 5 percent, with two seg- 
ments containing no barley at all. Because of the nature of the barley/other- 
small-grains labeling technique, the labeling accuracy for barley cannot be 
adequately tested if a reasonable amount of barley is not present. Therefore, 
in all of the subsequent discussions, barley is considered part of the small- 
grairs category, and labeling accuracies are evaluated for small -grains/ 
nonsmall -grains labeling only. 


The labeling accuracies for individual crops are shown in table 3-3. None of 
the nonsmall grain crops were consistently mislabeled; of the small -grain ' 
crops, only flax was incorrectly labeled more often that it was correctly 
1 abel ed. 


Note: This type of error for flax was observed during 

Phase III of LACIE (ref. 28) and the Transition 
Year (ref, 29). Although flax is not a small 
grain, its spectral signature is similar and is 
considered as grouped with small grains. 
Because there is so little flax, it is difficult 
to decide (on the basis of these and prior 
results) whether flax should be identified as a 
small grain or nonsmall grains. 
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TABLE 3-2.- SEGMENT-LEVEL RESULTS OF THE SHAKEDOWN TEST FOR THE OBJECTIVE 

REFORMATTED AND INTEGRATED PROCEDURES 


MM 


Correctly labeled dots, % 


■1 





Segment 

characteristics 

Procedure 


Barley 

Nonsmall 

grains 


Reformatted 

91 


93 

Z^% small grains 
(no barley) 

1542 

Integrated 

42 

- 

96 

3% other crops 


Reformatted 

86 

44 

88 

505^ small grains 
11% barley 

1584 

Integrated 

93.4 

45 

94 

Acquisitions deficient 
for barley 


Reformatted 

57 

- 

95 

75^ noncropland 
7% small grains 

1656 

Integrated 

52.6 


« 

97 

No barley 


Reformatted 

70 

81 

95 

38% small grains 
8% barley 

1664 

Integrated 

87 

54.5 

94 

27% other crops 

' 

Reformatted 

56 

36 

81 

25% small grains 
2% barley 

1811 







Integrated 

70 

0 

94 

40% other crops 


Reformatted 

76 

52 

91 


Overal ^ 

Integrated 

75 

55 

95 



^Segment 1514 was not processed during the Transition Year. 
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TABLE 3-3.- A^MLYST LABELING ACCURACY FOR INDIVIDUAL CROPS - 
OBJECTIVE REFORMATTED LABELING PROCEDURE, 

SHAKEDOWN TEST, PHASE 1 RESULTS 


Crop 

Number of 
dots labeled 

Crops correctly 
labeled, % 

Nonsmall grains 

Alfalfa 

58 

81 

Corn 

155 

78 

Sunflower 

109 

92 

Sugar beets 

14 

79 

Grass 

112 

93 , 

Hay 

137 

91 

Pasture 

539 

95 

Trees 

12 

83 

Water 

34 

94 

Nonagr 7 cultural 

111 

96 

Homestead 

23 

87 

Idle 

257 

89 

Small grains 

Spring barley 

111 

83 

Spring wheat 

443 

81 

Flax 

34 

41 

Spring oats 

92 

62 

Duram wheat 

16 

100 







3. 1.1. 3 Reformatted Procedure Cropland/Noncropland Labeling Accuracy , 

shakedown Test, Phase i 

The dots considered in evaluating the. corpland/noncropland labeling accuracy 
were those which had been determined pure by the analyst. Figure 3-1 is an 
illustration of the cropland/noncropland decision logic. The labeling 
accuracy for the cropland/noncropland decision logic is given in table 3-4(A). 
The labeling accuracy obtained as a function of the path taken through the 
decision logic is shown in table 3-4(B). None of the paths through the 
decision logic consistently produced wrong answers. 

When the decision logic method was used, 66 percent of the area within the 
segments was cropland. Labeling accuracy for the dots labeled cropland by 
decision 3 in the decision logic was lower than the accuracy for dots labeled 
noncropland, resulting in no problem as the incorrectly labeled cropland dots 
remained in the flow of the decision logic; they could be labeled nonsmall 
grains later. Therefore, if either cropland or noncropland were to have a low 
labeling accuracy, it is best for the dots labeled cropland to be mislabeled 
as they remain in the decision logic. No major problems surfaced when using 
the cropland/noncropland logic. 

3. 1.1. 4 Reformatted Procedure Small Grains/Nonsmal 1 Grains Labeling Accuracy. 
Shakedown Test, Fhase 1 

Table 3-5(A) shows the labeling accuracy for the small -grains/nonsmall -grains 
decision logic. The dots used in evaluating the small grains/nonsmall -grains 
labeling accuracy are those which were correctly identified as cropland by the 
analyst. The accuracy for this logic appears to be quite good, especially 
when there is agreement between the analysts on the label. In table 3-5(B), a 
wide variety of paths through the logic are used. None of the paths appear to 
produce consistently incorrect answers, which indicates that there are no 
’ major problems with the logic. 

As stated previously, there was not enough barley in these segments to deter- 
mine if the barley separation procedure was working properly; the accuracy in' 
separating barley from other small grains is presented in table 3-6. The dots 
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TABLE 3-4.- LABELING ACCURACY FOR CROPLAND/NONCROPLAND - 
OBJECTIVE REFORMATTED LABELING PROCEDURE, 
SHAKEDOWN TEST, PHASE 1 RESULTS 

(A) OVERALL LABELING ACCURACY 



Correctly labeled 


dots, % 

Cropland 

84 (90) 

Noncropland 

74 (82) 


•(B) ACCURACY BY PATH THROUGH THE DECISION LOGIC 

The numbers In parentheses reflect the percentage of 
labeled dots when both analysts agree<i on the label. 


Responses to cropland/ 
noncropland decision 
logic questions 
(See fig. 3-1) 

lA IB 1C 2 3 4 

Labeling 

decision 

Total 

dots 

labeled, 

Correctly 
label ed 
dots, % 

Crops which most 
frequently produce 
errors 

Y 


Crop 

52 

77 (85) 

Grass, pasture, 
nonagricultural , .idle 

Y 

- - N Y - 

Noncrop 

32 

82 (86) 

Alfalfa, corn, spring 
wheat, barley 

N 

Y . - - - 

Noncrop 

9 

84 (95) 

Spring wheat, barley 

i; 

N Y N N - 

Crop 

3 

90 (92) 

Idle 

N 

, N - - - 

Noncrop 


85 (-) 

Spring wheat 


Symbol det'nition: 

N = no 
Y = yes 
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TABLE 3-5.- LABELING ACCURACY FOR SMALL GRAINS/NONSMALL GRAINS 
DECISION LOGIC - OBJECTIVE REFORMATTED LABELING PROCEDURE, 
SHAKEDOWN TEST, PHASE 1 RESULTS 

(A) OVERALL LABELING ACCURACY 


Crop type 

Correctly labeled 
dots, % 

Small grains 
Nonsmall grains 



(B) ACCURACY BY PATH THROUGH THE DECISION LOGIC 

The numbers in parentheses reflect the percentage of 
labeled dots when both analysts agreed on the label. 


Responses to small grains/ 
nonsmall grains decision 
logic questions • 

(See fig. 3-1) 

iA IB 1C 2 3 4 

Labeling 

decision 

Total 

dots 

labeled, 

% 

Correctly 
labeled 
dots, % 

Crops which most 
frequently produce 
errors 

Y 

- 

Y ’ 

- 

Y 

SG 

22 

94 

(96) 

Corn, sunflower 

Y 

Y 

- 

N 

- 

NSG 

' 15 

73 

(78) 

Spring wheat 

Y 

Y 

- 

« 

Y 

SG 

13 

97 

(98) 

Hay 

Y 

- 

Y 

Y 

Y 

SG 

10 

94 

(95) 

Sunflower 

Y 

Y 

- 

- 

N 

NSG 

7 

84 

(100) 

Spring wheat 

Y 

Y 


Y 

Y 

SG 

6 

88 

(90) 

Corn 

Y 

- 

Y 

- 

N 

NSG 

6 

92 

(95) 

Spring wheat 

Y 

Y 

m 

Y 

- 

SG 

6 

95 

(100) 

- 

N 

- 

- 

- 

M 

NSG 

5 

81 

(87) 

Spring oats 

Y 

N 

N 

- 

- ’ 

NSG 

3 

95 

(100) 

- 


Symbol definition; 

NSG = non small grains 

SG = small grains 

Y = yes 

N = no 
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TABLE 3-6.- LABELING ACCURACY FOR SMALL 
GRAINS/BARLEY DISCRIMINATION - 
OBJECTIVE REFORMATTED PROCEDURE, 
SHAKEDOWN TEST, PHASE 1 RESULTS 


Crop type 

Correctly labeled 
dots, % 

Small grains 
(except barley) 

Barley 

9S (98) 
61 (54) 


used to determine this accuracy are those which were correctly labeled as 
small grains by the analyst. Only about half of the barley is correctly 
labeled, whereas almost all of the other small grains are labeled correctly. 

3.1.2 THE EXPANDED LABELING TEST (PHASE 2, PARTS 1 AND 2) USING 1979 DATA - 
DETAILED RESULTS 

The expanded labeling teat conducted during Phase 2 using the 1979 data set 
consisted of Parts 1 and 2. Part 1 was an analysis using the earlier, more 
subjective, integrated procedure SSG-0. The technology is the same which was 
used during LACIE and the TY projects. This methodology utilizes an analyst's 
experience and intuition for advantage in achieving best results. Part 2 was a 
test using the objective reformatted labeling procedure SS6-1 and applying it 
to nine segments in the USNGP, crop year 1979. Procedure SSG-1 utilizes 
specified decision logic for labeling. The results from the integrated test 
(Phase 2, Part 1) were used for con.parison with results from the objective 
reformatted labeling procedure test (Phase 2, Part 2). A comparison of the 
final results and ground truth for the objective reformatted and the 
integrated procedures is given in table 3-7. The following observations/con- 
clusions were noted when comparing the results of the respective procedures. 

a. The rate of segment process! bllity is higher using the integrated proce- 
dure. (Thirty-five segments were processed with equivalent accuracies'to 
the nine segments of the reformatted procedure. The integrated procedure 
can be executed without being limited by predefined acquisition . 
requi rements. ) 
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TABLE 3-7.- EXPANDED LABELING TEST RESULTS - PHASE 2, PARTS 1 AND 2 


The numbers in parentheses reflect the labeling 
accuracy when both analysts agreed on the label , 



Analyst labeling accuracy, % correctly labeled 

Crop 

category 

^ Phase 2, Part ,i 
Integrated procedure (SSG-0) 

Phase 2 
Reformatted pr 
9 seg 

!, Part 2 
ocedure (SSG-1), 
ments 

Same 9 
segments 
as used 
in SSG-1 

All segments 
available, 
35 segments 

All dots 

Pure dots 

Nonsmall 

grains 

94 

94 

75 (81) 

76 (81) 

Total small 





grains 

76 

73 

66 . (77) 

65 (76) 

Small grains 
except barley 

69 

66 

61 (73) 

61 (73) 

Barley 

59 

41 

15 (20) 

16 (21) 
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b. Although slightly lower, the labeling accuracies of the new reformatted 
procedure (SSG-1) compares favorably to accuracies of the integrated 
procedure (SSG-0) for small grains, with and without barley. 

c. The labeling accuracies of the reformatted procedure for nonsmall grains 
were low compared to results from the integrated procedure. 

d. Labeling accuracies for barley only are unacceptable with either 
procedure. 

e. The results for the objective reformatted procedure did not change when 
only pure dots were evaluated. 

The data reported in section 3.1.2 were extracted from working papers of E. R. 
Magness, Lockheed Engineering and Management Services Company, Inc., 1980. 

3.1.2. 1 Labeling Accuracy of the New Objective Reformatted Labeling 
Procedure, Phase 2, Part 2 

Labeling accuracies were determined for tv/o analysts and for each segment. 
Random dot proportion estimates were then tabulated for the best analysis of 
each segment, The evaluation was based on comparing analyst labels to the 
derived ground truth. The labeling accuracy for all dots labeled was 
71.6 percent for spring small grains and 82.5 percent for nonsmall spring • 
grains. These data are illustrated in figure 3-2. 

The labeling accuracy of spring small grains compares well with accuracies 
obtained in the past using purely interpretive methods (e.g., the integrated 
procedure). However, the labeling accuracy for nonsmall spring grains is much 
lower than accuracies obtained in the past. When viewed from the standpoint 
of proportion estimation for the nine segments, this low accuracy results in a 
relative overestimation of spring small grains by 23 percent. 

The clerical and implementation errors, errors related to procedural short- 
comings, and confusion errors between barley and other spring small grains are 
characterized in the following paragraphs. 
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ORIGiNAL PASE ES(’ 
OF POOR QUALITY 


OVERALL ACCURACY: Best of two analyses (accumulated) 


Ground Truth 


r 



S ^=-0.716 

209 _ n 17C 
TOT " 

Analyst 

N 0.284 

0.825 

Legend: 



N = Nonsmall spring grains 


S = Spring 

small grains 



Analyst 


Legend: 


Ground Truth 



S 


B 


N 


S 

269 _ 
W“ 

0.678 

-^= 0.586 

162 

TOT 

= 

0.136 

B 

15 _ 

0.378 

= 0.138 

47 


0.839 

OT7 ■ 

TOT 


N 

113 _ 
W 

0.284 

1 — 
<NJ 

• 

O 

II 

982 

TOT 

= 

0.825 


B = Barley 

N = Nonsmall spring grains 
S = Spring small grains 


Figure 3-2.- Accuracies of the objective reformatted labeling procedure 
Phase 2, Part 2 (nine segments of the 1979 crop year segments). 
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3. 1.2.2 Spring Small Grains Error Characterization. Phase 2. Parts 1 and 2 

Labeling error characterization for the objective reformatted labeling 
procedure and the integrated procedure are summarized’ in tables 3-8 and 3-9. 

The confusing of pasture with spring small grains and mixed or misregistered 
pixels contributed 30 percent and 25 percent, respectively, to the labeling 
errors when using the reformatted procedure (table 3-8). Similarly, 25 per- 
cent of the labeling errors for the integrated procedure were due to border/ 
edge pixels. Another 25 percent of the total error with the integrated proce- 
dure was attributed to the omission of late developing spring small grains 
(table 3-9). 

3. 1.2. 3 Characterization of Errors Common to Both Processings, 

Phase 2, Part 2 

For the nine segments, an average of 53 dots per segment were mislabeled by 
both analyst processings. This ranged from 8.2 percent to 19.1 percent of the 
dots in the nine-segment data set. (The actual number of dots in error ranged 
from 26 in segment 1676 to 64 in segment 1457.) 

3. 1.2. 3.1 Characterization of Errors Related to Implementation, Both 
Processings, Phase 2, Part 2 

A characterization of labeling errors which were related to the implementation 
of the reformatted procedure and common to both of the analysts was conducted. 
The results indicated an average of 14.3 dots per segment (3.9 percent) were 
incorrectly labeled because of segment procedure clarity or implementation 
probelms. These causes are listed below: 

a. Acquisition selection, 3.8 dots per segment - errors of omission 

b. Cropland/noncropland separation question implementation, 8.2 dots per 
segment - errors of commission. 

c. Clerical, 2.3 dots per segment - omission and commission errors. 

Labeling errors found to be caused by improper utilization of the acquisition' 
selection procedure resulted in errors of omission. This occurred in two 
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TABLE 3-8.- THE OBJECTIVE REFORMATTED PROCEDURE LABELING ERROR 
CHARACTERIZATION (9 SEGMENTS) - PHASE 2, PART 2 


Error causes, all dots 

Relative 

Importance, 

% 

Pasture: Commission to spring small grains 

30 

Mixed: Misregistered, omission and commission 

25 

Summer crop: Commission to spring small grains (sunflowers) 

10 

Crop calendar: Omission of spring small grains 

10 

Idle/fallow: Commssion to spring small grains 

5-10 

Procedure variability and clerical 

15-20 


TABLE 3-9.- THE INTEGRATED PROCEDURE LABELING ERROR CHARACTERIZATION - 

PHASE 2, PART 1 


Error causes 

Relative 

importance, 

% 

Border/edge: Omission and commission 

25 

Late development of spring small grains: Omission 

25 

Nonsmall spring grains follow small spring grains 
temporal trajectory: Commission 

15 

Signature confusion on a single date* Omission 

10 

Spring small grains follow nonsmall spring grains 
temporal trajectory: Omission 

10 

Early development of spring small grains: Qnission 

5 

Miscellaneous 

10 
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segments: the incorrect selection of the window 1 acquisition (too late) for 
segment 1394 and the use of an incorrect time period "A" acquisition for 
segment 1387. 

The analysts encountered difficulty in implementing the questions in the crop 
land/noncropland decision logic (figure 3-1). If the analysts had applied the 
questions more objectively, the errors probably would not have occurred. Most 
of the difficulty with the cropland questions occurred in segments located in 
North Dakota, agrophysical unit (APU) 21, where pasture was committed to 
spring small grains, 

3. 1.2. 3. 2 Characterization of Errors Related to the Procedure, Both 
Processings, Phase 2, Part 2 

The tree diagram and data, given in figures 3-3, 3-4, and 3-5, provide a 
breakdown of the procedure related to errors according to their cause,. 

3. 1.2. 4 Characterization of Errors Made in a Single Processing, 

Phase 2, Part 2 

For the nine segments, an average of 44.4 dots per segment were labeled incor- 
rectly by one of the two analysts, but not by both analysts. The number of 
errors that were not in common with the processings of the two-analysts pro- 
cessings ranged from a low of 25 in segment 1627 to a high of 69 in segment 
1571. These errors represented an error rate, ranging from 7.3 percent to 
17.5 percent for the segments. 

3. 1.2. 4.1 Characterization of Errors Related to Implementation, Single 
Processing, Phase 2, Part 2 

A characterization of labeling errors which were related to the implementation 
of the reformatted procedure and not in common to both analysts was conducted. 
The results indicated an average of 14.9 dots per segment in this category 
(3.7 percent) were incorrectly labeled because of segment procedure clarity or 
implementation problems. These causes are listed below. 

a. Acquisition selection, 0.8 dots per segment - errors of omission 
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The percentage of the errors made when answering each question In the 
cropland/noncropland decision logic shown In figure 3-1 Is given below. 


Crop 

1 

Question number and percentage of error 

lA 

IB 

■B 

2 

3 

mm 

Spring wheat 

0.6 


0.6 


4.0 


Oats 

0.3 




1.4 


Barley 





0.6 


Pasture/grass 

2.3 


1.7 


5.2 


Hay 




• 

2.6 


Idle/fallow 



■H 


2.0 


Total 

3.2 


IB 


15.8 



Figure 3-3.-Cropl and/noncropl and decision logic error characterization, 

both processings, Phase 2, Part 2. 
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The percentage of dots incorrectly identified by the logic for each 
window/period is given below: 


Crop 

Window/period and percentage of dots 
incorrectly identified 

1 

2 

3 

A 

4 

ALL 

Spring wheat 

2.6 

0.9 


2.0 

1.7 


Oats 

2.0 




2.3 


Barley 

1.1 






Flax 

0.4 






Pasture/grass 






8.9 

Hay 






5.4 

Idle/fallow 






8.3 

Sunflowers 






10.9 

Other 






4.6 

Total 

6.1 

0.9 


2.0 

4.0 

38.1 


Figure 3-4.- Green number decision logic error characterization.; 
both processings, Phase 2, Part 2. 
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Cause: 




^Pfl !19 wheat 4. 3^ 


Imagry comparison of 
analyst-selected pure dots 
and similar appearing dots 
with green numbers avail- 
_,ab1e 15,7^ 



Figure 3-5.- Imagery comparison error characterization, 
both processings, Phase 2, Part 2, 


3-20 


b. Cropland/noncropland separation question iinplefrientation, 7»4 dots per 
segment 

c. Clerical, 6.7 dots per segment 

Four out of seven acquisition selection errors were made in segment 1394; two 
errors were in segment 1457. All of the acquisition selection-related errors 
were errors of omission. Implementation difficulty with the cropland/noncrop- 
land separation questions occurred primarily in segment 1387, APU 19, of North 
Dakota (35 errors) and segment 1485, APU 17, of South Dakota (20 errors). The 
clerical errors were made primarily in segment 1457 (32 errors) and segment 
1394 (17 errors). 

3. 1.2. 4. 2 Characterization of Errors Related to the Procedure, Single 
Processing, Phase 2, Part 2 

The remaining valid procedure-related errors are Cjiven in figures 3-6., 3-7, 
and 3-8 according to the point in the procedure logic at which the errors 
occurred. 


3. 1.2.5 Barley/Other Spring Small Grains Error Characterization, 

Phase 2, Part 2 

Four hundrerd fifty-four spring small grain dots were labeled correctly as 
spring small grains by both processings. 

3. 1.2.5. 1 Dots Labeled Incorrectly in Both Processings 

Thirty dots (6.6 percent) were mislabeled at the barley and other spring small 
grains breakpoint in both processings. Table 3-10 indicates the number of 
mislabeled dots for each crop category according to the cause for mislabeling. 
Segment 1676 had no errors at this breakpoint. Among the other segments, 
there was fairly even distribution of the errors. In table 3-10, the barley 
committed to the spring small grains category and spring small grains com- 
mitted to the barley category errors almost balance each other (14 versus 16). 

Of additional significance is the fact that 24 out of 30 errors (80 percent) 
were caused by the dots falling within the wrong spectral distribution in the 
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Cause; Crop! and/noncropland 
' decis ion lo gic 35«2% 


^Pfj.D9 whea t 



The percentage of the errors made when answering each question in the 
cropland/noncropland decision logic shown in figure 3-1 is given below. 


Crop 

Question number and percentage of error 

lA 

IB 

1C 

2 

3 

4 

Spring wheat 

1.1 


0.8 

0.4 

0.7 


Oats 



0.35 


0.35 


Pasture/grass 





24 .0 


Hay 



0.4 


2.6 


Idle/fallow 



2.4 


0.8 


Other 




0.4 

0.7 


Total 

1.1 


4.15 

0.8 

29 .15 



Figure 3-6.- Cropland/noncropland decision logic error characterization, 

single processing, Phase 2, Part 2. 
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Spring wheat 1 , 1 % 


Omission 1.1% 


use: Green number logic 48.9%* 


Commission 47.6% 


Pasture/grass 28% 

Hay j.7% 

•Idl e/fallow 5.2% 


Corn 1.5% 


’Sunflowers 4.5% 
Other 4.5% 


The percentage of the errors made in each window/period is given below. 


Crop 


Spring wheat 

Pasture/grass 

Hay 

Idle/fallow 

Corn 

Sunflowers 

Other 

Total 


Window/period and percentage of errors 


1 2 3 A 4 An 


0 .4 0 .4 0 .4 



0 .4 0 .4 



Figure 3-7.- Green number decision logic error characterization 
single processing, Phase 2, Part 2. 














om&mi f ® 

OF POOI^ quality 


Cause : 


Imagery comparison of analyst- 
selected pure dots and 
similarly appearing dots with 
green numbers available 16.1% 


Spring wheat 4.8% 



Figure 3-8.- Imagery comparison error characterization, 
single processing. Phase 2, Part 2. 


TABLE 3-10.- NUMBER OF MISLABELED DOTS BY CATEGORY AND CAUSE 

IN BOTH PROCESSINGS 


Cause of mislabeling 

Barley committed to the 

other small grains 
_ 

Other spring small 
grains committed to 
the barley category 

Spring 

wheat 

Oats 

Flax 

Total 

Direct interpretation 

2 

0 

0 

2 

0 

Decision line 

4 

0 

0 

4 

0 

Early development 

2 

6 

0 

8 


Nonseparation 

- 

- 

- 

- 

16 

TOTAL 

8 

6 

0 

14 

16 
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scatter plots. These errors are caused either by early development of spring 
wheat and oats or by nonseparation of barley from the other spring small 
grains even with correct application of the procedure. 

3. 1.2. 5. 2 Errors Made in Only One Processing 

Twenty-eight dots (6.2 percent) of the spring small grain dots were labeled 
correctly by one analyst and incorrectly by the other analyst at the barley 
and small grains breakpoint. Table 3-11 indicates the number of mislabeled 
dots for each crop category according to the cause for mislabeling. Segments 
1387, 1627, and 1676 had no errors. Twenty-two errors were in the following 
segments: 

a. Segment 1394: 6 dots, apparent early Wheat maturity 

b. Segment 1457: 7 dots, no separation of barley 

1 dot, an incorrect interpretation by analyst 

c. Segment 1658: 8 dots, apparent early wheat maturity * 

1 dot, an incorrect interpretation by analyst 

Of significance is the fact that direct analyst interpretation was responsible 
for eleven (39 percent) of the errors made by a single analyst. Note also 
that it was this 39 percent that contributed most to the imbalance between the 
barley commission errors of spring grains and spring grains/barley, 

3. 1.2. 6 Consistency of Labeling Between the Two-Analyst Processings Using the 
Objective Reformatted Label i ng Procedure (SSG-l) In Phase 2, Part 2 

For all dots labeled on the nine segments, 78 percent were labeled consist- 
ently. (Consistent means that both analysts assigned the same label to a 
given dot, but not necessarily the correct label.) The accuracy of the dots 
labeled consistently by both analysts by crop category is in table 3-7. The 
labeling accuracy for total small grains for the consistently labeled dots 
averaged 77 percent. 
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TABLE 3-U.- NUMBER OF MISLABELED DOTS BY CATF^ORY AND CAUSE 

IN ONE PROCESSING 


Cause of mislabeling 

Barley committed to the 
other small grains 

Other spring small 
grains committed to 
the barley category 

Spring 

wheat 

Oats 

Flax 

Total 

Direct interpretation 

10 

0 

0 

10 

1 

Decision line 

2 

0 

0 

2 

1 

Early development 

8 

0 

0 

8 

- 

Nonseparation 

- 

- 

- 

- 

6 

TOTAL 

20 

0 

0 

20 

8 
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3 . 2 MACHINE PROCESSING/CLASSIFICATION TEST RESULTS 


Results of the machine processing/classification test, based upon ground-truth 
input labels, are: 

a. A significant increase in the precision of segment proportion 
estimates was obtained by CLASSY stratification (table 3-12). 

(1) This was the first time a machine processing technique had 
performed oetter than the technique of using simple random sampling 
and making the proportion estimate by relative count. 

(2) It requires three times as many labeled pixels for a randomly 
sampled segment in order to achieve the same proportion estimation 
precision as when CLASSY stratification is used. 

b. Segment proportion estimation bias and MSE are significantly reduced 
by machine processi ng/CLASSY stratification when compared with the 
results from random sampling (table 3-12). 

c. There is not a significant difference in the performance of the three 

machine allocation and estimation techniques: (1) proportion 

allocation/ relative count, (2) proportional allocation/Bayes 
estimator, and (3) Bayes sequential allocation/Bayes estimator. 

Summarized in table 3-12 are the biases of proportion estimates, standard 
deviations of estimate errors, and MSE's for 35 segments using the four 
procedures with dot labels from analyst interpreters (integrated procedure) as 
input. The errors are shown in figure 3-9. 

Machine processing/classification test results are in addendum 2, volume II, 
of this document and also in reference 3. 

3.2.1 PROPORTIONAL ALLOCATION/RELATIVE COUNT ESTIMATOR FINDINGS 

The proportional allocation/relative count estimator provided a significantly 
less biased estimate and produced less variable errors than did random 
sampling. The fact that the errors were less variable indicated that the 
clustering algorithm had been effective. 
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TABLE 3-12.- PROPORTION ESTIMATION RESULTS WITH GROUND TRUTH 

AND ANALYST LABELS 



Ground truth labels 

Analyst labels 

Proportion estimation technique 

Bias, % 

Standard 

deviation 

MSE 

1 

Bias 

Standard 

deviation 

MSE 

Random sample/relative count 

-2.5 

6.9 

53 

-5.7 

7.7 

90 

Proportional allocation/ 
relative count, CLASSY 
strati fi cat ion 

0.0 

4.0 

16 

-4.0 

6.2 

53 

Proportional allocation/ 
Bayes estimator, CLASSY 
strati fi cation 

0.5 

3.8 

14 

-3.5 

6.0 

47 

Bayes sequential allocation/ 
Bayes estimator, CLASSY 
stratification 

0.4 

4.7 

22 

-2.7 

6.8 

52 


















3.2.2 PROPORTIONAL ALLOCATION BAYESIAN ESTIMATOR FINDINGS 


Since clustering was effective, the next step was to determine the effect of a 
Bayesian estimator. With the proportional allocation/Bayesian estimator, a 
cluster-level Bayesian estimator was used instead of a vegetative count esti- 
mator. It had been hypothesized that the proportional allocation/Bayesian 
estimator would provide improved proportion estimates over the proportional 
allocation/relative count estimator because prior knowledge of cluster puri- 
ties was being considered. As hypothesized, there seemed to be improved 
precision, but the difference was small (table 3-12). These results were 
encouraging because they supported the expectation that Bayesian estimation at 
the cluster level would provide greater precision (although slightly biased 
results) over maximum likelihood estimation. 

3.2.3 BAYESIAN SEQUENTIAL ALLOCATION/BAYESIAN ESTIMATOR FINDINGS 

The final technique was the Bayesian sequential allocation/Bayesian estimator. 
The results showed it to be the least biased technique when ^analyst interpreter 
labels were used as input (table 3-12). 

This had been hypothesized since the dots were allocated to clusters one at a 
time, with the intention of minimizing the MSE. Although it produced the 
least biased results, the Bayesian sequential allocation/Bayesian estimator 
produced more variable results than did proportional allocation. This was a 
disturbing observation. That the Bayesian sequential .allocation/Bayesian 
estimator produced more variable results than did the proportional al-location/ 
Bayesian estimator was caused (in part) to a decreased overall labeling 
accuracy. 

Further testing indicated that with a small sample of dots (50), proportional 
allocation is the sampling method that produces the most precise and reliable 
estimtes. However, if a large enough sample size were taken, the same 
precision could be obtained by random sampling without the need of clustering 
information. 
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3.2.4 RECOMMENDATIONS FOR MACHINE PROCESSING/CLASSIFICATION 


While automatic labeling would provide large samples at relatively low costs, 
it is only a goal. With large samples, these clustering procedures do not 
seem to provide much improvement in proportion estimation. However, it is not 
recommended that effective clustering algorithms be discarded. Neither should 
efforts in proportion estimation techniques be defaulted to random sampling. 

An effective procedure using clustering information is available for use in 
testing and for future development. It should be noted that automatic label- 
ing has only recently been developed. It is therefore recommended that these 
proportion estimation techniques be maintained, particularly the proportion 
allocation/Bayesian estimator, because it provided the greatest precision. 

A recommendation from this exploratory experiment was that the proportion 
allocation/Bayesian estimator estimation procedure be considered the baseline 
for the 1981-1982 Spring Small Grains Pilot Experiment. Further exploratory 
testing needs to be conducted for other crops of interests such as corn and 
soybeans. 

3.3 CROP DEVELOPMENT STAGE MODEL TEST - SUMMARY 

■■■■ ' ' " ■ ' ' " ' ' " ■■■' ' * 

The result from the test of planting date models follows: the Feyerherm mo.del 

is significantly better than the normal model for predicting both spring wheat 
and barley planting dates (figures 3-10 and 3-11, respectively). 

Results from the test of the wheat phenological development stage models are: 

a. There are no significant differences between the three models 
(original Robertson and the two improved versions) in estimating the 
development stages from tillering to ripening. 

b. The improved Robertson models, versions 1 and 2, appear to estimate 
the late heading and ripening stages of wheat more accurately than 
the original Robertson model. 

The result from the test of the barley phenological development stage models 
follows: the Robertson spring wheat models performed better than the Williams 
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Figure 3-11.- An illustrated comparison of the Feyerherm and 
normal models for predicting barley planting dates. 


barley model. None of the models predicted the wheat /barley separation period 
very accurately. 

3.3.1 DETAILED RESULTS FOR PLANTING DATE MODELS 

Both the Feyerherm and the normal models produce median planting date 
estimates at the segment level. The performances of the models for the spring 
wheat fields and the barley fields were evaluated separately. 

A histogram of the distribution of errors, measured in days for the Feyerherm 
versus the normal planting date models applied to spring wheat fields, gives 
an indication of the bias associated with the models. Both distributions 
appeared normal, the differences being the locations of the midpoints of these 
distributions. The Feyerherm model has a positive displacement, whereas the 
normal model has a negative displacement. This indicates that the normal 
model is very early compared to the ground-truth median planting dates, where- 
as the Feyerherm model is slightly late. (Based on reports jointly published 
by the USDA and National Oceanic and Atmospheric Administration of the USDC in 
the Weekly Weather and Crop Bulletin, the 50 percent planting date of -spring 
wheat in North Dakota for 1979 was 13 days late. Thus, the normal model 
performed as expected.) 

The statistics on tne errors, measured in days for the Feyerherm model versus 
the normal model applied to spring wheat, are summarized in table 3-13. The 
sign test shown is based on the absolute magnitude of the error and gives the 
percent of times one model is closer to the ground truth than the other model r 
Table 3-13 indicates that, on the average, the Feyerherm model is 3.9 days 
late compared to the observed planting date, whereas the normal model is, on 
the average, 10.4 days early compared to the observed planting date. Statis- 
tically, the sign test indicates that the Feyerherm model is significantly 
better than the normal mode'i <is\t the 6-percent level of significance. The over- 
all statistics indicate that the Feyerherm model is closer to the ground truth 
than the normal model in predicting spring wheat planting dates for the year. 


3-35 


TABLE 3-13.- COMPARISON OF ERRORS IN PLANTING DATE MODELS 
APPLIED TO SPRING WHEAT FIELDS 


• 

Feyerherm model 

Normal model 

Number of segments (n) 

49 

49 ■ 

Mean error (In days) 

+3.9 

-10.4 

Standard deviation (In days) 

7.0 

7.50 

Median error (in days) 

+4.0 

-11,0 

Sign test (%) 

75.5 

22.4 

iZ% tied) 




For the distribution of error, measured In days for the Feyerherm versus the 
normal planting date models applied to barley fields, both distributions 
appear normal. However, the Feyerherm model midpoint has a positive displace- 
ment, whereas the normal model has a negative displacement. This indicates 
that the two models are, on the average, late and early compared to the 
ground-truth median planting dates, as seen for barley fields. 

The statistics on the error, measured 1n days from the Feyreherm model versus 
the normal model applied to barley fields, are summarized in table 3-14. Note 
that, on the average, the Feyerherm model Is 2.9 days later than the observed 
planting date, whereas the normal {55>del Is, on the average, 10.9 days earlier 


TABLE 3-14.- COMPARISON OF ERRORS IN PLANTING DATE MODELS 
APPLIED TO BARLEY FIELDS 


. 

Feyerherm model 

Normal model 

Number of segments (n) 

44,0 

44.0 

Mean error (in days) 

+2.9 

-10.9 

Standard Deviation (In days) 

11,48 

9.55 

Mediih error (In days) 

+4,5 

-11.5 

Sign test 

63 . 6 

36.4 
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than the observed planting date. The sign test Indicates that the Feyerherm 
model Is better than the normal model, though not statistically significant at 
the 5-percent level of significanpe. The overall statistics Indicate that, 
for this year, the Feyerherm model Is better than the normal model for pre- 
dicting barley planting dates. Addendum 3 in volume II of this document con- 
tains detailed Information pertaining to the results for planting date models. 

3.3.2 APPROACH FOR EVALUATION OF CROP DEVELOPMENT STAGE MODELS 

The three Robertson models and the Williams barley model were started using 
the ground-truth median planting dates for spring wheat and barley fields as 
input to the models. They were evaluated on their ability to accurately 
predict median crop development stages for spring wheat and barley between 
stages 2.0 and 6.0, which are the emergence through ripe stages. 

In an attempt to determine if the models performed differently during various- 
periods of the growing season, the models were evaluated at five ranges of the 
growth stages as shown below. 


1. 

Stage 2.0 to 

2.9: 

emergence to prejointing 

2. 

Stage 3,0 to 

3.9: 

jointing to preheading 

3. 

Stage 4.0 to 

4.9: 

heading to presoft-dough 

4. 

Stage 5.0 to 

5.9: 

soft-dough to preripening 

5. 

Stage 6.0: 


ripe 


In addition, the overall performance was tested for the entire growing season 
from stage 2.0 to stage 6.0. 

3.3.3 CROP DEVELOPMENT STAGE MODEL RESULTS APPLIED TO SPRING WHEAT FIELDS 

Figure 3-12 contains scatter plots of the median-predicted development stages 
versus the observed median development stages for models RO, Rl, and R2. The 
letters represent the number of data points falling on the character (A = 1, 

B = 2, etc.). The common trend on all three plots is for the predicted growth 
stage to converge on the 1-1 line, indicating that the performance of all 
three models is improving with time through the growing season. In - 
figure 3-12, one can See that model RO is progressing faster than models Rl 
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and R2 by noting that 13 ground-truth observations are off-scale and greater 
than stage 6,0 (i.e., swathed and harvested). 

The statistics on the errors between the observed stages and the predicted 
stages that were applied to spring wheat at various intervals throughout the 
growing season are sunmiarized in table 3-15. The errors are the differences 
between the predicted stages and the observed stages and should give an 
indication of the amount of bias associated with each of the models. An 
average positive error would indicate that the model is ahead of the ground 
truth, whereas an average negative error would indicate that the model was 

behind the ground truth. In addition, the absolute value of the error was 

ranked on a scale of 1 to k, where k is the number of models being compared 

with each other (in table 3-15, k = 3). The sum of the various ranks 
associated with each model was then utilized in a Friedman nonparametric test 
of ranks (ref. 30) to determine if any one model produced better results 
consistently. 

Table 3-15 shows that there were no significant differences between any of the 
three models when evaluating the overall performance from ground-truth 
stages 2.0 to 6.0. The range of the mean error for the three models was two- 
tenths of a stage, and the Friedman T-statistic indicates that there is no • 
significant difference between the models at the 95-percent confidence level. 

For stages 2.0 to 2.9, there was a marginal difference between the three 
models. It is apparent that R1 is the worst performer of the three model c at_ 
this stage interval, as indicated by the statistics on the errors and the 
observed sum of the ranks. From stages 3.0 to 3,9, there was a significant 
difference between the models. Model RO appeared to be the best at this stage 
interval. From stages 4.0 to 4.9, there was no significant difference between 
the models. For stages 5.0 to 5.9, there was a significant difference between 
the models, and R1 appeared to perform the best within this stage interval.. 
Finally, at stage 6.0, there was a significant difference between the three 
models. Model R2 appeared to perform the best of the three models. _ At 
ground-truth stage 6.0, the mean and standard deviation have not been 
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! TABLE 3-15.- COMPARISON OF ROBERTSON MODELS APPLIED TO SPRING WHEAT 


Ground-truth 

range 

Statistic , 

Robertson 0 

Robertson 1 

Robertsofi 2 

2.0 - 6.0 

Mean error 

0.0 

0.2 

0.2 

Enti re 

s'ro ' ‘ 

0.53 

0.48 

0.46 

growl ng 

Median error 

0.0 

0.1 

0.2 

season 

ERank observed 

100.21 

97.08 

96.71 


Friedman's T-statistic; 0.15 (not significant) 

2.0 - 2.9 

Mean error 

0.9 

1.0 

0.9 


STD 

0.25 

0.28 

0.25 


Median error 

0.9 

1.0 

0.9 


ERank observed 

25.00 

37.75 

27.25 


Friedman's T-statistic: 6.17 (significant) 

3.0 - 3.9 

Mean error 

0.3 

0.7 

0.4 


STD 

0.26 

0.32 

0.26 


Median error 

0.3 f' 

0.7 

0.4 


ERank observed 

42.42 

95.25 

66.33 


Friedman's T-‘statisttc: 41. 

17 (significant) 

CTl 

• 

1 

O 

• 

Mean error 

-0.2 

0.1 

0.1 


STD 

Ci.26 

0.27 

0.31 


Median error 

-0.2 

0,1 

0.0 


ERank observed 

89.67 

70.75 

79.58 


Friedman's T-statistic; 4.48 (not significant) 

5.0 - 5.9 

Mean error 

-9.2 

0.0 

0.1 


STD 

0.42 

0.27 

0.33 


Median error 

-0.2 

0.0 

0.2 


ERank observed 

109.45 

■' 66.60 

93.95 


Friedman's T-statistic: 20. 

92 (significant) 

6.0 

Mean error 



— 


STD 


-- 

— 


Median error 

-0.5 

-0,4 

-0.3 


ERank observed 

50.0 

48.4 

33.5 


Friedman's T-statistic: 24. 

07 (significant) 


At 95-percent confidence level, Friedman's T-statistic critical value = 5.-99 
At 99-percent confidence level, Friedman's T-statistic critical value = 9.’21 
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displayed, as they are not valid. The observations obtained beyond stage 6.0 
were beyond the range of the model's abilities of prediction and, therefore, 
were not valid, 

3.3.4 CROP DEVELOPMENT STAGE MODEL RESULTS APPLIED TO BARLEY FIELDS 

Figure 3-13 contains scatter plots of the median-predicted development stage 
for model R2 and the Williams barley model versus the observed median develop- 
ment stage. The letters represent the number of data points falling on that 
character. At first glance, there is no apparent difference between the two 
models, although the barley model appears to be more dispersed about the 
1-1 line than. model R2. More significant is the fact that 33 observations are 
beyond 6.0, indicating that the barley model is progressing faster than the 
spring wheat model. 

In table 3-16 are the statistics on the errors between the median ground-truth 
stage and the model -predicted median stage applied to barley at various stage 
intervals through the growing season. Table 3-16 indicates that there was a 
significant difference between the models for the overall performances from 
stages 2.0 to 6.0. The barley model is significantly worse than at least one 
of the spring wheat models. 

From stage 2.0 to 2.9, there were marginal differences between the models. 
Model RO appeared to perform the best of the four models as indicated by the 
error statistics and the observed sum of the ranks. For stages 3.0 to 3.9, 
there was a significant difference between the models. Model RO appeared to - 
be the best of the four models. From stages 4.0 to 4.9, there were no 
significant differences between the models. They appeared to be nearly 
identical at this stage interval. For stages 5.0 to 5.9, there was a 
significant difference between the models. Model R1 appeared to perform the 
best. At stage 6.0, there were no significant differences between the models, 
and model R2 appeared to perform the best. 
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TABLE 3-16.- COMPARISON OF ROBERTSON AND WILLIAMS MODELS APPLIED TO BARLEY 


Ground-truth 

range 

Statistic 

Rob^ertson 0 

Robertson 1 

Robertson 2 

Wi 1 1 i ams 
barley 

2.0 - 6.0 

Mean error 

-0.2 

0.0 

0.0 

0.4 

Enti re 

STD 

0.67 

0.60 

0.61 

0.60 

growi ng 

Median error 

-0.2 

0.0 

0.0 

0.0 

season 

SRank observed 

117.67 

96.96 

98.58 

126.79 


Friedman's T-statistic: 8.74 (significant) 

2.0 - 2.9 

Mean error 

1.0 

1.1 

1.0 

1.2 


STD 

0.32 

0.37 

0.33 

0.35 


Median error 

1.1 

1.2 

1.1 

1.2 


sRank observed 

22.33 

33.50 

24.67 

39.50 


Friedman's T-statistic: 9.49 (significant) 

3.0 - 3.9 

Mean error 

0.3 

0.4 

0.4 

0.6 


STD 

0.32 

0.38 

0.36 

0.42 


Median error 

0.2 

0.4 

0.3 

0.5 


SRank observed 

50.58 

90.67 

65.08 

’ 113.67 


Friedman's T-statistic: 43.79 (significant) 

' 

4.0 - 4.9 

Mean error 

-0.3 

-0.1 

-0.2 

0.1 


STD 

0.32 

0.34 

0.38 

• 0.52 


Median error 

-0.3 

0,0 

-0.1 . 

0.2 


TRank observed 

89.42 

62.67 

74.92 

79.0 

' ' 

Friedman's T-statistic: 7.18 (not significant) 


5.0 - 5.9 

Mean error 

-0.5 

-0.3 

-0.2 

0.1 


STD 

0.57 

0.45 

0.54 

0.59 


Median error 

-0.6 

-0.2 

-0.2 

0.3 

' 

ERank observed 



129.93 

70.67 

95.10 

_ 114.30 


Friedman's T-statistic: 28. 

68 (significant) 

- 

6.0 

Mean error 

MM 





STD 

— 

-- 


— 


Median error 

-0.9 

-0.7 

-0.6 

>0.0 


ERank observed 

48.0 

35.0 

26.5 

50,5 


Friedman's T-statistic: 14, 

31 (significant) 



At 95-percent confidence level, Friedman's T-statistic critical value = 7.-S2. 
At 99s“percent confidence level, Friedman's T-statistic critical value = 11.34. 
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GROUND OBSERVED BARLEY CRO? STAGE 

Figure 3-13.- Scatter plots of predicted versus ground-observed barley crop stages for 
Williams barley model and the Improved Robertson, version 2, spring wheat model. 


3.3.5 RECOMMENDATIONS FROM CROP DEVELOPMENT STAGE MODEL TESTS 

Based on the results to date, it is recommended that the Feyerherm planting 
date model be utilized for both spring wheat and barley. It appears that the 
improved Robertson model, version 2, is the more useful for predicting spring 
wheat and barley development stages. However, the model is not adequate to 
determine window 3 of the reformatted procedure, which is used to separate 
barley from spring wheat. Further research on biowindow 3 is required if 
accurate results are to be obtained for identifying this window. 


4. CONCLUSIONS AND RECOMMENDATIONS 


The spring small grains labeling procedures evaluation results are listed 

below. 

a. Reformatted labeling (SSG-1) results are comparable to those of the 
integrated analysis procedure SSG~0. 

b. The SSG-1 procedure is conducive to automation. 

c. Error sources in the SSG-1 procedure are easily identified and quantified 
due to the tree-structured design of the procedure. 

d. Improvements to the SSG-1 labeling logic are required to eliminate the 
confusion of pasture and spring small grains. 

e. Additional criteria for defining acceptable SSG-1 Landsat acquisitions for 
processing are required. 

f. Labeling is consistent, using -the reformatted SSG-1 labeling procedure. 

The results from the SSG-1 wheat/barley separation evaluation follow: 

a. Labeling accuracy was approximately 50 percent in low-density barley 
segments, 

b. Because high-density barley segments were not available, the procedure was 
not adequately evaluated. 

c. Crop development stage models were insufficient for selecting the 
wheat/barley separation acquisition. 

The machine processing/classification procedure results indicate the 

following. 

a. CLASSY stratification improved the precision of the proportion estimation 
procedures. 

b. Estimation bias and MSE were significantly reduced over random sampling 
for the first time ever. 
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The crop development stage model test results indicate the following. 

a. The Feyerherm planting date model performs batter than the normal mods! 
for both spring wheat and barley. 

b. The performance of all three versions of the Robertson spring wheat model 
is similar. 

c. The performance of the Robertson and Feyerherm models appears to be 
satisfactory for integration into automated labeling procedures; however, 
further evaluation is recommended. 

In summary, the results of the e><pl oratory experiment indicate it to be a 

strong potential for establishing the basis for a highly efficient technology 

for evaluation in a foreign environment. 

Three recommendations follow: 

1. A pilot experiment on spring ’small grains in the USNGP and Canada should 
be conducted to further develop, test, and evaluate the technology prior 
to initiating a foreign pilot experiment. 

2. The technology focus should be directed towards techniques for efficient 
• area estimation and procedures for sensitivity analysis of spring small, 

grain area estimation. 

3. The expected assessment performance in foreign countries should be 
considered. 
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